Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions

Abstract Idiomatic expressions (IEs), characterized by their non-compositionality, are an important part of natural language. They have been a classical challenge to NLP, including pre-trained language models that drive today’s state-of-the-art. Prior work has identified deficiencies in contextualized representation stemming from the underlying compositional paradigm representation. In this work, we take first-principles approach build idiomaticity into BART using adapter as lightweight non-compositional expert trained on idiomatic sentences. The improved capability over baselines (e.g., BART) is seen via intrinsic and extrinsic methods, where idiom embeddings score 0.19 points higher homogeneity for embedding clustering, up 25% sequence accuracy processing tasks IE sense disambiguation span detection.